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Origins of IV&V within NASA 







NASA's IV&V Program: established in 1993 
Founded under the NASA Office of Safety and Mission 
Assurance (OSMA) as a direct result of recommendations 
made by the National Research Council (NRC) and 
the Report of the Presidential Commission on the Space 
Shuttle Challenger Accident. 
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The Need for IV&V 



NASA Decides That A Software Error Doomed The 
Mars Global Sui'veyor Spacecraft 

By Keith Cowing Posted Wednesday, January 10, 2007 



During a meetmg of the Mars Explorat Peter b. de a 

Group Meeting in Washington Dc, vest 
McNamee. Mars Exploration Program 
the recent failure of the Mars Global Sim’eyor tMtJ 
spacecraft. 


Software Glitch Blamed for Turning Satellite 
Into Space Zombie 


Office of the 


Chief Engineer q C 




Software Glitch Means Loss of NASA's D< 
Comet Probe 

■ ^ir 

timothy posted about 4 months ago | from Taco Cowboy 

Taco Cowboy writes "'NASA is ce 
comet D*~Qbe after a suspected sc 



Reports 

Text Size Q Q 

Technical Excellence Initiative 
NASA Study on Flight Software Complexity 

In 2007 the NASA Office of Chief Engineer commissioned a multi-Center study of 
the growth in flight software size and complexity in NASA space missions. The 


Inquiry Board Traces Ariane 5 Failure to Overflow Error 


Readers of SIAM News may remember that on Jime 4, less than a minute into its first flight, the Frendi rocket Ariane 5 self-i 
des etudes spatiales) and ESA (the European Space Agency) to investigate the failure was chaired W applied mathematician 


Developing complex, safety and mission-critical software systems 
is inherentiy chailenging, and that creates risk. 
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What is IV&V? 



Independent Verification and 
Vaiidation (iV&V) is an objective 
examination of safety and mission 
critical software processes and products 

independence: 3 key parameters: 

• Technical Independence 

• Managerial Independence 

• Financial Independence 

NASA IV&V perspectives: 

• Will the system’s software. . . 

• Do what it is supposed to do? 

• Not do what it is not supposed to do? 

• Respond as expected under adverse 
conditions? 



Systems Engineering: Determines if 
the right system has been built and 
that it has been built correctly 


w 

IV&V Technical Approaches: 

• Aligned with IEEE 1012 

• Captured in a Catalog of 
Methods 

• Spans the full project lifecycle 


IV&V Assurance Strategy 

The IV&V Project’s strategy for providing mission assurance 
Assurance Strategy is driven by the specific needs of an individual project 

Implemented via an Assurance Design 
Communicated via Assurance Statements 
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What is IV&V? (continued) 



• The IV&V Assurance Strategy is the selection and implementation of 
IV&V validation and verification processes 

- Implementation of the IV&V processes are driven by the IV&V Project’s risk 
assessment and unique characteristics 

- The Assurance Strategy is tailored to the needs of the individual projects 

• The validation process provides empirical evidence that engineering 
products: 

- Satisfy system requirements allocated to software 

- Solve the right problems 

- Satisfy the intended use and user needs in expected operational environments 

• The verification process provides empirical evidence that 
engineering products: 

- Conform to requirements (for example: for correctness, completeness, 
consistency, accuracy) during all life cycle phases (requirements, design, code, 
test) 

- Satisfy standards and best practices 

- Establish a basis for assessing the completion of each life cycle phase, and 
initiating other life cycle phases 
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What is IV&V? (continued) 



• IV&V processes include assessments, analyses, evaluations, 
reviews, inspections, and testing of software artifacts during 
the entire development lifecycle that create evidence 

- Evidence is used to formulate recommendations that improve the quality (or 
reliability) of the system software 

- Evidence is used to make conclusions about the quality (or reliability) of the 
system software 

- Evidence is used to gain insight into the technical progress 

- Evidence is used to judge how thorough you've critiqued the system 

• How much evidence ^ it is a trade-off between criticality of the 
system being acquired/deployed 

- Life-sustaining subsystems would warrant an evidence package that clearly & 
objectively shows the software will operate safely (or clearly shows that it 
won't) 

- Data management subsystems may warrant less of an evidence package 

• The amount of evidence needed determines the rigor of the analysis 

- Analytical Rigor is the type and amount of IV&V processes to use for analysis 
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Establishing the IV&V 
Assurance Strategy 


• The IV&V Program assesses the system to determine: 

- The inherent risk associated with the system capabilities 

- The role of software in those capabilities 

- Which software elements of the system warrant IV&V analysis 

- Software elements are generally the focal point of IV&V analyses; however, 
other lifecycle artifacts (for example: concept documentation, system design, 
etc...) are utilized to inform lower-level analyses 

• Our process is called “Portfolio Based Risk Assessment” (PBRA) 

- Results in scores for impact (a measure of the effect of a problem) and likelihood 
(the potential for the existence of errors) for each system capability and software 
element 

- Enables informed decision making regarding: 

• What parts of the system should IV&V work on 

• What analytical rigor should IV&V apply (for example: dynamic 
analysis should be conducted to thoroughly test the implementation of 
the protocol used for communications) 
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Establishing the IV&V 
Assurance Strategy (continued) 
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Impact 

Subsystem 1 - do not recommend IV&V 
Subsystem 2 - recommend IV&V utilizing Static Analysis 
Subsystem 3 - recommend IV&V utilizing Dynamic Analysis 
Subsystem n ... 


less ^ Amount of Rigor & Evidence Needed > more 

Manual Analysis 


Static Analysis 


Dynamic Analysis 


Formal Analysis 

SMEs conduct formal 
or informal inspections & 
evidence is recorded simply 
as issues 

SMEs evaluate structure 
& content using various 
perspectives supported 
by CASE tools. Evidence is 
recorded as issues & 
supplemented with coverage 

SMEs execute system & 
evaluate results. Evidence is 
recorded more thoroughly as 
to make the case for what 
works and what are limitations 

SMEs apply formalisms 
& mathematical rigor 
to prove existence or 
absence of critical 
properties 
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Implementing the IV&V 
Assurance Strategy 



• IV&V Assurance Strategy is implemented through the Assurance 
Design 

• The Assurance Design specifics the Technical Reference, inputs, analysis 
techniques, and objective evidence necessary to achieve the IV&V Project's 
Objectives 

• Like the Assurance Strategy, the Assurance Design is specific to the needs of an 
individual project 

• Constructed to allow the IV&V Project to generate evidence to assure the 
critical capabilities and mitigate system risk 

• Areas of risk identified in the PBRA are key inputs into the development of 
the Assurance Design 

• Assurance Statements are utilized to communicate the results of 
the implementation of the IV&V Assurance Strategy 

• A statement of the assurance that is being provided (or intended to be provided) 
by IV&V to a stakeholder or stakeholders on a system or subsystem 

• Assurance statements are typically formulated at the beginning of a IV&V 
Project and refined as necessary throughout execution 
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Tools for Implementing the 
IV&V Assurance Strategy 


• NASA’s IV&V Program strives to continually develop new 
capabilities to support the execution of the IV&V Assurance 
Strategy 

- IV&V Techniques are documented in a Catalog of Methods (CoM) 

- Techniques are continually refined and tailored to the needs of the 
projects 

• To maintain relevance, the IV&V Program selectively invests 
in new technologies necessary to assure NASA’s safety and 
mission critical software 

- NASA’s IV&V Program is advancing the state of the practice in 
Cybersecurity / Information Assurance and Independent Testing 

• Advanced techniques and capabilities are being developed to 
enable the program to keep pace with current development trends 
and emerging risk factors 

• Information Assurance and Independent Testing are becoming an 
increasingly prominent component of IV&V Project’s Assurance 
Strategies 
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Cybersecurity / Information Assurance 


Ensuring Mission and Safety Critical Software and Systems Operate Reliably, Safely, 

and Securely 

Threat and Risk Assessment Vuinerabiiitv Assessment / 


• FISMA Compliance 

• Life-cycle 

• Provide mission security assurance 
throughout design, development, 
implementation, operation, maintenance, 
and disposition 

• Assessment and Authorization (A&A) 

• Authority to Operate (ATO) 

Penetration Testing 

• Implementation of Security Controls 

• Monitoring of Security Controls 

• static Code Analysis (SCA) 

IV&V In-Phase lA Support 

CvberLab 

• Build security in “from the ground up.” 

• Security Architecture Verification 

• IV&V Methods 

• Component of ITC JSTAR Lab 

• Virtualized servers 

• Penetration Test tools 

• Cybersecurity Knowledge Base 

• Cybersecurity Training Program 

• Mission System Virtualization and Testing 


11 


Independent Testing 


Develop, maintain, and operate adaptable test environments for NASA’s IV&V Program 
that enable the dynamic analysis of software behaviors for multiple NASA missions 


Simulation 

Automation 

• Functional Software-only Simulators 

• NASA Operational Middleware (NOS) 

o Common emulation software 
o Middleware 

• Spacecraft Simulators 

o Ground systems, instruments, spacecraft 
dynamics 

• Small Sat 

• Integrate many technologies to create solutions 

• Simulation Verification 

• Increase Testing 

o Unit Testing 
o System Testing 

• Automated Installations and Simulator Deployments 

Testing 

Virtualization 

• Provide evidence-based assurance to customer 

• Risk-focused independent testing 

• Focused on testing adverse conditions 

o Fault injection, back-to-back scenarios, etc. 

• Heavy reliance on virtualization technologies 
o Development 
o Simulator Releases 
o Rapid Deployment 
o Evaluation Environments 
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Summary 
Benefits of IV&V 



• Yields higher confidence that delivered products are error free and meet 
the user needs . 

• Increases likelihood of uncovering high-risk errors early in the 
development lifecycle . 

- Allows time for the design team to evolve a comprehensive solution rather than forcing 
them into a makeshift fix to accommodate deadlines 

• Delivers ongoing status indicators and performance reporting to decision 
makers (e.g. program managers). 

- The customer is provided an incremental preview of system performance with the 
chance to make early adjustments . 

• Reduces the need for rework from the developing contractor thereby 
reducing total costs to programs and projects . 

• Facilitates the transfer of system and software engineering best practices . 


IV&V leads to higher quality products, reduced risk, greater 
insight, reduced cost, and knowledge transfer. 
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QUESTIONS? 
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IV&V Services 




■ IM «fclW« 

CCP DSCQVR ECTP GPM GOES-R GSDO InSight 



ISS JWST JPSS MAVEN MMS MPCV OSIRIS-REx 



SPP SLS SGSS 


i lV&V plays a key role in a number of high-profile NASA and non- 

NASA missions. 
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Generic Look at IV&V 
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Criticality Analysis {identify most critical areas of the system} 









